Skip to content

Fix cross-bundle run_job_task refs pointing at non-existent nodes - #37

Closed
matthewmoorcroft wants to merge 2 commits into
unify/reconcile-internalfrom
pr/cross-bundle-run-job-refs
Closed

Fix cross-bundle run_job_task refs pointing at non-existent nodes#37
matthewmoorcroft wants to merge 2 commits into
unify/reconcile-internalfrom
pr/cross-bundle-run-job-refs

Conversation

@matthewmoorcroft

@matthewmoorcroft matthewmoorcroft commented Sep 1, 2026

Copy link
Copy Markdown
Member

⚠️ Draft — initial port. This change is rebuilt onto the unified base as-is. It predates the engine → sources/adf module restructure, so it still needs adaptation to the new layout (and a green test run) before it is mergeable.

Closes #29

What

ExecutePipeline emits run_job_task.job_id = ${resources.jobs.X.id}, which only resolves when X is a job in this bundle. In a multi-pipeline migration each ADF pipeline becomes its own bundle, so a reference to a sibling pipeline points at a node that doesn't exist here and bundle deploy fails with no such node "resources.jobs.X".

New _rewrite_cross_bundle_run_job_refs runs in write_bundle before the YAML is written: it rewrites run_job_task refs to out-of-bundle jobs into ${var.X} and registers X in _cross_bundle_variables (which the existing _build_databricks_yml loop already declares). The operator supplies the numeric job id at deploy via --var, as SETUP.md documents. Recurses into for_each_task bodies.

Relationship to the lineage epic (#24)

This is the stopgap ("FLOWX-7") that #24 (ordered cross-pipeline deploy/run from control lineage) is designed to build on — #24 keeps this ${var.X} rewrite as the fallback for unresolved callees and layers an ordered orchestration bundle on top once #23's lineage.control_edges exist. This fix makes multi-pipeline deploy work today; it does not conflict with or block #24.

Scope

src/flowx/bundler/dab_writer.py only (the cross-bundle rewrite; the task-value dependency fix is a separate PR).

Testing

make test full unit suite green; test_bundler.py + test_bundle_invariants.py pass.

This pull request and its description were written by Isaac.

@matthewmoorcroft
matthewmoorcroft marked this pull request as ready for review September 1, 2026 12:16
@matthewmoorcroft Isaac
Point uv.lock at public PyPI
0411be6 
Rewrite internal package-proxy URLs (pypi-proxy.dev.databricks.com) to
pypi.org / files.pythonhosted.org so public CI resolves deps. Same pinned
versions and hashes; matches main.

Co-authored-by: Isaac <no-reply@databricks.com>
@matthewmoorcroft

matthewmoorcroft commented Sep 9, 2026

Copy link
Copy Markdown
Member Author

Closing as obsolete. The problem this targets (#29 — cross-bundle run_job_task refs pointing at non-existent resources.jobs.X nodes) is already solved in the unify/reconcile-internal base by _rewrite_cross_bundle_job_references (dab_writer.py), which rewrites out-of-bundle refs to ${var.<job>_job_id} using the pydabs-inclusive _known_bundle_job_keys set — covered by the passing test_cross_bundle_job_reference_uses_declared_job_id_variable.

This PR's _rewrite_cross_bundle_run_job_refs was an initial port predating that base work. Because the canonical pass runs first, this second pass only ever matched refs it had correctly left as ${resources.jobs.X.id} — i.e. legitimate in-bundle pydabs jobs — rewriting them to ${var.X} and breaking test_python_resource_job_reference_remains_a_resource_substitution and test_shared_airflow_bundle_namespaces_pydabs_hooks_and_jobs. Removing it leaves dab_writer.py byte-identical to the base (verified: full unit suite 1194 passed / 2 skipped). No code to salvage; #29 is already resolved.

@matthewmoorcroft matthewmoorcroft mentioned this pull request Sep 9, 2026
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@matthewmoorcroft